Real-time priority-based media communication

ABSTRACT

Representative embodiments are disclosed of a real-time priority-based communication system and method for communicating media streams made up of multiple media message sub-streams, the communication system comprising a chunk configurator for dividing media message sub-streams into chunks, a state machine configured to translate between information regarding the media message sub-streams and the chunks and state associations to the information, the state associations written into a header for each of the chunks, a queue for holding the chunks waiting to be transmitted, and a processor for executing a scheduling algorithm, wherein the scheduling algorithm determines ones of the chunks in the queue to transmit next.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.14/084,160, filed Nov. 19, 2013, which is a continuation of U.S.application Ser. No. 13/620,658, filed on Sep. 14, 2012, and issued asU.S. Pat. No. 8,626,942 on Jan. 7, 2014, which is a continuation of U.S.application Ser. No. 13/304,311, filed on Nov. 24, 2011, and issued asU.S. Pat. No. 8,285,867 on Oct. 9, 2012, which is a continuation of U.S.application Ser. No. 13/250,199, filed on Sep. 30, 2011, and issued asU.S. Pat. No. 8,301,796 on Oct. 30, 2012, which is a continuation ofU.S. application Ser. No. 12/554,775, filed on Sep. 4, 2009, and issuedas U.S. Pat. No. 8,065,426 on Nov. 22, 2011, which is a continuation ofU.S. application Ser. No. 11/841,517, filed on Aug. 20, 2007, and issuedas U.S. Pat. No. 7,587,509 on Sep. 8, 2009, which is a continuation ofU.S. application Ser. No. 10/366,646, filed on Feb. 13, 2003, and issuedas U.S. Pat. No. 7,272,658 on Sep. 18, 2007, the benefit of priority ofeach of which is claimed hereby, and each of which are incorporated byreference herein in its entirety.

TECHNICAL FIELD

The present invention relates, in general, to media communications, and,more specifically, to a system and method for a real-time priority-basedcommunication system.

BACKGROUND OF THE INVENTION

In the early years of the Internet, its primarily use was for thereliable transmission of data with minimal or no delay constraints.Transmission Control Protocol (TCP), of the TCP/Internet Protocol (IP)protocol suite, was designed for this type of delay independent datatraffic. TCP typically works well in this context where the reliabilityof the packet delivery is much more important than any packet delays. Inorder to achieve this reliability, TCP sets up a connection at both endsand attaches a header to each packet that contains the source anddestination ports as well as the sequence number of the packet and othersuch administrative information. The destination typically receives anumber of TCP packets before sending an acknowledgement to the source.If the acknowledgment fails, the source will generally presume thepackets were lost and retransmit the “lost” packets. While this processensures reliable delivery, packets may be delayed which, in multimediastreams, may cause noticeable and unacceptable degradation of quality inthe multimedia playback.

An alternative transmission protocol in the TCP/IP protocol suite isUser Datagram Protocol (UDP). Unlike TCP, UDP is connectionless andunreliable, meaning that it does not establish a connection at both endsand does not include a resource for resending lost packets. Instead, theUDP packets are sent out with a packet header that typically includesonly the source and destination ports along with a 16-bit segment lengthand 16-bit checksum for minimal error detection. Because UDP does notinclude the additional administrative information, it generally makes nodelivery guarantees, offers no flow control, and performs only minimalerror detection. As such, UDP has useful timing characteristics forreal-time audio or video transmission, where the delivery of everypacket is not as important as the timely delivery of packets. UDP wasgenerally used as the early transport protocol for real-time multimediaapplications because it typically offers these beneficialcharacteristics for delay-sensitive data delivery. However, by itself,UDP usually does not provide any general purpose tools that may beuseful for real-time applications.

In response to the limitations of UDP, Real-time Transport Protocol(RTP) was developed to operate as a thin layer on top of UDP to create ageneralized multipurpose real time transport protocol. An RTP fixedheader may generally include: a 7-bit payload type field for identifyingthe format of the RTP payload; a 16-bit sequence number which isincremented by one for each subsequent RTP data packet transmitted; a32-bit timestamp that corresponds to the time that the first RTP datapacket was generated at the source; a 32-bit synchronization sourceidentifier which is a randomly generated value that uniquely identifiesthe source within a particular real-time session; as well as otheradministrative information. With this information, RTP provides supportfor applications with real-time properties including timingreconstruction, loss detection, security, and content identificationwithout the reliability-induced delays associated with TCP or the lackof timing information associated with UDP.

Real-Time Control Protocol (RTCP) works in conjunction with RTP toprovide control support to the application for maintaining the RTPsession. RTCP generally performs four functions: (1) providinginformation to the application regarding the quality of transmission,such as number of packets sent, number of packets lost, interarrivaljitter, and the like; (2) identifying the RTP source through atransport-level identifier, called a canonical name (CNAME), to keeptrack of the participants in any particular RTP session; (3) controllingthe RTC P transmission interval to prevent control traffic fromoverwhelming network resources; and (4) conveying minimal sessioncontrol information to all session participants. The RTCP packets aretypically transmitted periodically by each participant in an RTP sessionto all other participants. Therefore, RTCP provides performance anddiagnostic information that may be used by the application.

One of the major problems associated with streaming multimediainformation arises in attempting to stream the media through firewalls,Network Address Translation (NAT) devices, and the like. The majorpurpose of firewalls is to prevent unauthorized and/or hostile access toa computer system or network. As such, firewalls are generallyconfigured with strict rules specifying specific, static ports throughwhich desired and/or authorized data traffic can pass, while blockingundesirable data. The majority of all IP protocols use RTP fortransporting the media streams. RTP is built over UDP, which generallyhas no fixed ports associated with it. Thus, there is no guarantee thata port associated with the incoming RTP/UDP stream will be allowedthrough the firewall. Moreover, each media stream typically has multiplechannels, which generally requires its own opening through the firewall.This means that for the media stream to traverse the firewall, thefirewall will have to open many UDP openings for each call session,which defeats the purpose for the firewall in the first place.

NAT devices are used to translate an IP address used within one networkto a different IP address known within another network. NAT devicestypically maintain a map of addresses within an “inside” network. Anycommunications directed to users within the inside network usually passfirst through the NAT device for translation to the inside address.Thus, users within the inside network may see out, but outside users canonly typically communicate with the inside users through the NATdevice's translation. NAT devices may allow a network to support manymore users or clients than it has fixed IP addresses. The NAT device maybe addressed from the outside using the few fixed IP addresses, yetservice many other address within the inside network.

Another problem with the existing streaming protocols is the amount ofheader information attached to any given piece of data on the stream. Asmentioned above, UDP and TCP messages contain considerable headerinformation concerning the timing, sequence, data type, and the like.Because multiple streams are typically running at once, each piece ofdata generally has a stream ID to tell the destination which stream anyparticular piece of data belongs to. In a situation where an audiostream is established where the message type is constant for a period oftime and the formatting of the message type requires a constant numberof bits per message, the header information on the type, length, and thelike congests the stream with useless information, thus taking away fromthe maximum available data bandwidth.

BRIEF SUMMARY OF THE INVENTION

Representative embodiments of the present invention are directed to areal time priority-based communication system for communicating mediastreams made up of multiple media message sub-streams, the communicationsystem comprising a chunk configurator for dividing media messagesub-streams into chunks, a state machine configured to translate betweeninformation regarding the media message sub-streams and the chunks andstate associations to the information, the state associations writteninto a header for each of the chunks, a queue for holding the chunkswaiting to be transmitted, and a processor for executing a schedulingalgorithm, wherein the scheduling algorithm determines ones of thechunks in the queue to transmit next.

Additional representative embodiments are directed to a method forcommunicating media streams, comprised of a plurality of sub-streams, ina real-time priority based communication protocol, the method comprisingdividing each one of the plurality of sub streams into a plurality ofchunks, assigning states to each of the plurality of chunksrepresentative of information regarding the associated sub-streams andthe associated media streams, writing the states into a header of eachof the plurality of chunks, queuing the plurality of chunks fortransmission, and transmitting ones of the plurality of chunks accordingto a scheduling algorithm.

Further representative embodiments are directed to a computer programproduct having a computer readable medium with computer program logicrecorded thereon, the computer program product comprising code forbreaking down each one of a plurality of media sub-streams making up amessage stream into a plurality of associated chunks, code forimplementing a state machine that processes state assignments for eachof the plurality of associated chunks representative of informationregarding the media sub-streams and the message streams, code forwriting the state assignments into a header of each of the plurality ofassociated chunks, code for building a queue for the plurality ofassociated chunks waiting for transmission, and code for transmittingones of the plurality of associated chunks according to a schedulingalgorithm.

The foregoing has outlined rather broadly the features and technicaladvantages of the present invention in order that the detaileddescription of the invention that follows may be better understood.Additional features and advantages of the invention will be describedhereinafter which form the subject of the claims of the invention. Itshould be appreciated by those skilled in the art that the conceptionand specific embodiment disclosed may be readily utilized as a basis formodifying or designing other structures for carrying out the samepurposes of the present invention. It should also be realized by thoseskilled in the art that such equivalent constructions do not depart fromthe spirit and scope of the invention as set forth in the appendedclaims. The novel features which are believed to be characteristic ofthe invention, both as to its organization and method of operation,together with further objects and advantages will be better understoodfrom the following description when considered in connection with theaccompanying figures. It is to be expressly understood, however, thateach of the figures is provided for the purpose of illustration anddescription only and is not intended as a definition of the limits ofthe present invention.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

For a more complete understanding of the present invention, reference isnow made to the following descriptions taken in conjunction with theaccompanying drawing, in which:

FIG. 1 is a conceptual block diagram illustrating a data packet which ispart of a typical media stream;

FIG. 2 is a conceptual block diagram illustrating a typical messagestream comprising multiple message packets;

FIG. 3 is a conceptual block diagram illustrating one example embodimentof a chunk configured according to the real-time priority-basedcommunication system of the present invention;

FIG. 4 is a conceptual block diagram detailing a chunk control byte;

FIG. 5A is a conceptual block diagram illustrating a chunk streamcomprising multiple chunks;

FIG. 5B is a block diagram illustrating a state machine configured for arepresentative embodiment of a real-time priority-based communicationsystem, as described herein, receiving a Type 0 chunk;

FIG. 5C is a block diagram illustrating a state machine configured for arepresentative embodiment of a real-time priority-based communicationsystem, as described herein, receiving a Type 3 chunk;

FIG. 5D is a block diagram illustrating a state machine configured for arepresentative embodiment of a real-time priority-based communicationsystem, as described herein, receiving a Type 3 chunk;

FIG. 5E is a block diagram illustrating a state machine configured for arepresentative embodiment of a real-time priority-based communicationsystem, as described herein, receiving a Type 3 chunk;

FIG. 6 is a flowchart disclosing representative steps of an examplescheduling algorithm used in representative embodiments of the presentinvention;

FIG. 7A is a conceptual block diagram illustrating a real-timecommunication system configured using a representative embodiment of thereal-time priority based protocol of the present invention in apeer-to-peer application;

FIG. 7B is a conceptual block diagram illustrating a real-timecommunication system configured using a representative embodiment of thereal-time priority based protocol of the present invention in aclient-server application;

FIG. 8 illustrates a computer system adapted to use the presentinvention; and

FIG. 9 illustrates a sequence of interleaved audio and video chunks in areal-time priority-based communication system configured according tovarious embodiments of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Before beginning a discussion of the various embodiments of the presentinvention, it may be helpful to review how media streams are carriedthrough transmission.

FIG. 1 is a conceptual block diagram illustrating data packet 10 whichis part of a media stream. Data packet 10 not only comprisesdata—payload 104, but also comprises header information that is used toassemble and play the media stream at the destination entity. The headerinformation typically comprises message stream ID 100, message type ID101, time 102, and length 103. Message stream ID 100 identifies theparticular stream that data packet 10 is a part of. In a typicalmultimedia application, such as an audio/video conference, for example,the outgoing stream may be a first stream, while the outgoing videostream may be a second stream. There are possible scenarios that includea large number of streams and sub-streams going back and forth to thedifferent participants over a single connection. In a typicalapplication a media stream is comprised of multiple media message substreams that make up the different aspects of the main media stream. Inorder to deal with the many different streams, each stream is designatedwith a unique ID, such as message stream ID 100. Thus, when data packet10 enters the destination, the media application reads message stream ID100 and assembles data packet 10 into the appropriate message stream. Ina typical application, media stream ID 100 may comprise a 32-bitdesignator.

The header information may also comprise message type ID 101 messagetype ID 101 identifies the specific format of data packet 10. Forexample, message type ID 101 may identify data packet 10 as an audiodata format (or a particular type of audio data format, such as a wavedata format (.wav), MP3, and the like), a video data format (or aparticular type or video data format, such as a motion picture expertsgroup (MPEG)-2 or -4, a MICROSOFT™ WINDOWS MEDIA AUDIO™ (.wma), and thelike), or a pure data format. By designating the type of data format,the destination application will likely know how to play or display theincoming media stream.

Because a media stream generally comprises a group of packetstransmitted over some network, it is important for the destinationapplication to be able to assemble and play the data packets in orderand according to the timing of the original image/media. Therefore, theheader information also includes time 102, which is typically a timestamp for the recording or sampling of the particular packet. Thedestination application may use time 102 to synchronize each of thepackets that it receives in the stream. The application will then knowto play a packet with time 102 designating 1020 before a packet withtime 102 designating 1040. The time stamp may be a real-time, starttime, or the like. The purpose is to determine when the message occurredso that things may be synchronized later during broadcast or playback.

Length 103 is also a typical piece of header information that identifiesthe length of data packet 10. By designating the specific length of datapacket 10, the destination application knows when the current datapacket or media message has been fully received.

It may often be the case where a message stream carries data packets inwhich message stream ID 100, message type ID 101, and length 103 may bethe same value. Where packets are received consecutively, message streamID 100 should generally be the same. Furthermore, if the stream is anaudio stream, for example, message type ID 101 should also generally bethe same. Length 103 can vary to some extent, however, some mediaformats, such as constant bit-rate MP3, Nellymoser, and the like, maydivide all packets into the same length. Therefore, in such cases,length 103 will also be the same for each packet.

FIG. 2 is a conceptual block diagram illustrating typical message stream20 comprising message packets 21-24. Message packet 21 begins with theheader information: message stream ID 100 of 12345, message type ID 101of 8 (for audio), time 102 of 1000 and length 103 of 32 bytes. Thisheader information is important for the destination application toassociate the message packets with the correct stream and play them inthe right manner and in the right order. Message packet 22 also includesthe header information: message stream ID of 12345, message type ID 101of 8, time 102 of 1020, and length 103 of 32 bytes. As may be seen inFIG. 2, message packet 22 contains much of the same header informationas message packet 21. In fact, the only difference in the two messagepackets is the time stamp of time 103: message packet 21 has time 103 of1000, while message packet 22 has time 103 of 1020. A difference of 20seconds. Message 23 and 24 also comprise the same information for theheader information of message stream ID 100, message type ID 101, andlength 103, with time-stamps, time 102, differing by exactly 20 secondsper packet. The repeated information in message packets 21-24 clogs theavailable bandwidth of the entire system.

In an ideal system, it would be most efficient to send a data payloadwithout any header information. However, without header information, itis practically impossible to assemble and playback data packets in amedia stream with the current technology, let alone do so efficientlyFIG. 3 is a conceptual block diagram illustrating one example embodimentof chunk 30 configured according to the real-time priority-basedcommunication system of the present invention. A c-hunk is a piece of amessage stream that has been broken down or divided by a chunkconfigurator or some other message processing Chunk 30 comprises chunkcontrol byte 300 and data payload 301 chunk control byte 300 preferablycontains much of the header-type information necessary for managing thechunk stream.

It should be noted that in additional embodiments of the presentinvention, version byte 302 may be written/transmitted to provideinformation on the particular version of the transport protocol beingused. The inclusion of this information usually allows easiermaintenance of future protocol versions and backwards compatibility. Inoperation, only a single version byte would likely be necessary perconnection. Therefore, version byte 302 would likely not add anyoverhead on a per chunk basis.

In attempting to reduce the overhead of stream messages, certainassumptions may be leveraged into bandwidth savings. One such assumptioninvolves the necessity of a 32-bit message stream ID. Statistically,there are usually less than sixty-one (61) message streams (havingunique message stream IDs) in a server at any given time for any givenclient. Sixty-one (61) message streams may be uniquely represented inonly 6 bits. Thus, having a 32-bit message stream ID is typically awaste of space. FIG. 4 is a conceptual block diagram detailing chunkcontrol byte 300. Chunk control byte 300 comprises a chunk identifier,such as chunk stream ID 40 and chunk type ID 41. While the statisticssuggest that there are a maximum of sixty-one (61) message streams in aserver at any one time for a particular client, artificially limitingthe maximum number of streams does not present itself as an efficientmeans for obtaining bandwidth at the expense of stream capacity.Instead, chunk stream ID 40, as configured according to the embodimentillustrated in FIG. 4, provides a mechanism for supporting 65,597 totalsimultaneous streams (using a range of 3 to 65,599).

Chunk stream ID 40 comprises bits 0-5 of chunk control byte 300. Asprovided in Table 1, when the value of chunk stream ID 40 lies between 3and 63 in certain embodiments of the present invention, chunk stream ID40 is assigned at face value. Statistically, this situation will occurmost often, resulting in a 6-bit chunk stream ID. However, when thevalue of chunk stream ID 40 is 0 in certain embodiments of the presentinvention, a second byte is added to chunk stream ID 40 that representsthe range of 64 to 319. When the value of chunk stream ID is 1 incertain embodiments of the present invention, two additional bytes, asecond byte and a third byte, are added to chunk stream ID 40 thatrepresent the range of 320 to 65,599. Therefore, the embodimentillustrated in FIGS. 3 and 4 supports up to 65,597 streams, while stillconserving bandwidth on a statistically large number of messages. Achunk stream ID 40 value of 2 in certain embodiments of the presentinvention represents that the chunk contains a low-level protocolmessage, which are used to control administrative aspects of theembodiment illustrated in FIGS. 3 and 4.

TABLE 1 CHUNK STREAM ID (BITS 0-5 OF CHUNK CONTROL BYTE) Value Meaningof Value 3-63 Chunk Stream ID is the face value of 3-63. 0 There isanother byte that represents the range 64-319 (Second byte + 64). 1There are two more bytes representing the range 320-65,599 ((Thirdbyte)*256 + (Second byte + 64)). 2 The Chunk is a Low-Level ProtocolMessage.

The configuration of the real-time priority-based protocol illustratedin FIGS. 3 and 4 advantageously leverages the statistical maximum ofsixty-one (61) streams. Because there are usually less than sixty-one(61) streams in the server at any given time for a client only the 6bits of chunk stream ID 40 will generally be sent. Thus, the 32-bitheader for message stream IDs of the current technologies is preferablyreduced to only 6 bits for the majority of data streams. In rare cases,in which the number of streams exceeds 61, the additional bytes ofinformation may be included to support those extra streams.

The configuration of chunk stream ID 40 in certain embodiments of thepresent invention, as provided in Table 1, demonstrates implementationof a state machine having different state associations using its first 2bits to address many different levels of streams. In furtherance of thestate machine concept implementing the embodiments of the presentinvention, chunk type ID 41 provides four states/types for classifyingeach chunk message. Unlike the media type indicator of message type ID101 (FIG. 1) that identifies the format of the data payload, chunk typeID 41 identifies the type or purpose of the chunk.

As provided in Table 2, bits 6 and 7 of chunk control byte 300 comprisechunk type ID 41. When the value of chunk type ID 41 is 0 in certainembodiments of the present invention, the chunk state or type indicatoris a new/reset message type (Type 0) or indicates a new stream. Thissignifies the first message sent on the chunk stream in certainembodiments of the present invention. As is described in Table 2, a Type0 message may comprise additional bytes of information that include thetypical header information of the current stream methods, such asmessage length, message type ID, and message stream ID. Instead of usinga timestamp, however, the embodiment of the present inventionillustrated in FIGS. 3 and 4 uses a time delta (Δ), or the change intime from the previous message. The time-delta (Δ) is essentially anabsolute time because it is usually added to ‘0’ time. Because the Type0 chunk includes a considerable amount of information, it will generallybe larger than the subsequent chunks in the stream.

When the value of chunk type ID 41 is 1 in certain embodiments of thepresent invention, the chunk state or type is a new media indicator of aspecified new time change/time delta, length, and any new message typeID (Type 1). This state may signify a new chunk that is measured usingthe new specified time delta (i.e., the time stamp change from the lastmessage) and the specific length. Thus, a subsequent message in a chunkstream that belongs to the same chunk stream ID, but that is a differenttype/format and a different time delta than the previous chunk only hasto include header information on the message type ID, time delta, andthe length. A Type 1 chunk may, therefore, have fewer bytes of headerinformation than the Type 0 chunk, because the state machineimplementing the illustrated embodiment of the present invention hasbeen set with certain information by the Type 0 chunk that remainsunchanged for the Type 1 chunk.

A value for chunk type ID 41 of 2 in certain embodiments of the presentinvention represents the chunk state or type for a new message definedonly by a single piece of header information, such as the time delta(Type 2) in a new time indicator. A Type 2 chunk preferably has all ofthe same information from the previous chunk except possibly for achange in the time delta. Therefore, the state machine may only need touse the new time delta to change to the proper new state.

The least amount of overhead or header information in the representativeembodiment illustrated in FIGS. 3 and 4 accompanies the Type 3 chunk incertain embodiments of the present invention, when the value of chunktype ID 41 is 3. A Type 3 chunk is a continuation indicator. Acontinuation indicator chunk preferably uses the same message stream ID,the same message type ID, the same time change/time delta, and the samelength of the previous chunks. Therefore, the state machine implementingthis embodiment of the present invention is preferably already set tothe appropriate state. One design goal may be to optimize the streamingprocess to get the chunks to Type 3 chunks as soon as possible.Therefore, Type 3 chunks that are essentially continuations of a samemessage are transmitted with a 0-length header.

The 0-length header of the Type 3 chunks is especially interesting whenthe Type 3 chunk represents the beginning of a new message. Thisscenario may occur when two consecutive chunks use the same messagestream ID, the same message type ID, the same time change/time delta,and the same length as the previous chunk. In this case, a new messageis enabled without any required header information, a 0-length header.

TABLE 2 CHUNK TYPE (BITS 0-5 OF CHUNK CONTROL BYTE) Chunk Type BitsDescription 00 (Type 0) Begin new message. Reset all information. Type 0is used when the time moves backwards or for the first message sent onthe chunk stream. A Type 0 chunk message resets the time to 0 beforeadding the supplied time delta (Δ). A Type 0 chunk message will containthe following header information: 3 bytes - message time delta (Δ) 3bytes - message length 1 byte - message type ID 4 bytes - message streamID 01 (Type 1) Begin new message using specified Δ, new length, and/ornew message type ID. A Type 1 chunk message will generally contain thefollowing header information: 3 bytes - Δ 3 bytes - new message length 1byte - message type ID 10 (Type 2) Begin new message using specified Δ.A Type 2 chunk message will generally contain the following headerinformation: 3 bytes - Δ 11 (Type 3) Continuation (possibly start newmessage). A Type 3 chunk message will generally contain no headerinformation, but just the raw data/payload.

FIG. 5A is a conceptual block diagram illustrating chunk stream 50comprising chunks 500-503. Chunk 500 is a Type 0 chunk with a chunkstream ID of 4. Because chunk 500 is a Type 0 chunk, it includesadditional header data 504. The additional data includes the messagestream ID (12346), the message type ID (9), the message length (435),and the time delta (20). This header information is set into the statemachine of the present invention. Chunk 501 is a Type 3 chunk belongingto the same chunk stream ID 4. Because chunk 501 belongs to the samemedia stream ID and is the same type and length as chunk 500, the stateset by the extra header information of chunk 500 already defines how tohandle chunk 501. According to the teachings of the present invention,chunk 501 only needs to designate the chunk stream ID and chunk type.Therefore, only a single byte of overhead is added to the data payloadof chunk 501. Chunks 502 and 503 are also Type 3 continuation chunksthat require only an additional byte of header information to adequatelyhandle those chunks.

FIG. 5B is a block diagram illustrating state machine 51 configured fora representative embodiment of a real-time priority-based communicationsystem, as described herein, receiving a Type 0 chunk. State machine 51sees that chunk 500 is a Type 0 chunk and expects each of the statevariables to be set with the header information contained in chunk 500.In some embodiments, state machine 51 may also include message count 504that keeps track of the progress of each receiving message. Chunk 500brings 128 bytes of data payload. Therefore, using the message lengthparameter, state machine 51 can subtract 128 bytes from the totallength, 435 bytes, to hold the remaining message expected length of 307bytes in message 504. The state machine illustrated in FIG. 5B is usedin both the send direction and the receive direction. As chunks arriveat the receiver system, the state machine is used in reverse totranslate the state assignments necessary to play the data stream chunk.Therefore, the state machine for the real-time priority-basedcommunication system may be located at least at the sending entity andthe receiving entity.

FIGS. 5C-5E are block diagrams illustrating state machine 51 receivingType 3 chunks. In FIG. 5C, state machine 51 reads that chunk 501 is aType 3 chunk, and therefore, expects that only the chunk type and chunkstream ID state variables will be changed. Because chunk 501 couldbelong to a different sub-stream than that of chunk 500, the chunkstream ID is used in the Type 3 continuation chunks. The remaining statevariables set by chunk 500 remain the same. Message 504 is changed againto reflect the 128 bytes of payload data delivered by chunk 501. 179bytes now remain in the message. In FIG. 5D, state machine 51 reads thatchunk 502 is also a Type 3 chunk, and therefore, also expects that onlythe chunk type and chunk stream ID state variables will be changed. Theremaining state variables set by chunk 500 continue remain the same.Message 504 is again changed to reflect the 128 bytes of payload datadelivered by chunk 501. 179 bytes now remain in the message. FIG. 5Ealso illustrates state machine 51 receiving a Type 3 continuation chunk.With this receipt because state machine 51 knows the maximum chunk datapayload length is 128, state machine 51 knows that this will be the lastchunk for this particular message, as reflected in message 504 zeroingout. The next chunk will, by process of elimination, have to be for anew message (or be a low-level protocol message).

When compared to the header information included with each data packetin media stream 20 (FIG. 2), instead of resending the same header witheach data packet, the teachings of the present invention preferablyallow this information to be sent only once, with subsequent chunksleveraging the state machine in order to reduce the overhead sent witheach subsequent chunk. The amount of repetitive information is,therefore, favorably reduced, increasing the available bandwidth overthe connection.

Referring back to Table 1, when the value of chunk stream ID 40 is 2, incertain embodiments of the present invention, the state is designed tochange for the system to expect a low-level protocol message. Table 3describes two low-level protocol messages that may be implemented intovarious embodiments of the present invention. In such embodiments, amessage ID of 1 represents a change to the maximum chunk size. Theparameter passed with such a message ID may be an integer representingthe size for all future chunks. Another message ID, shown as message ID2 in Table 3, may allow for an abort signal to be sent to the networkcorresponding to a particular chunk stream ID. For example, if acommunication session is in mid-stream when the transmitting endshuts-down, or other such event, the abort message would generallyinform the network that it should not expect any further chunks, thus,aborting the stream. Many other optional low-level protocol messages mayalso be provided for in various embodiments of the present invention.The two messages provided for in Table 3 are merely examples.

TABLE 3 LOW-LEVEL PROTOCOL MESSAGES Message Message ID Message DataDescription Set 1 4 bytes representing The integer represents Chunk aninteger. Bytes are the new maximum chunk Size sent in “network” size forall future chunks. order (i.e., order of significance) Abort 2 4 bytesrepresenting The integer represents a Message an integer. Sent in chunkstream ID. If that “network” order. chunk stream is currently waiting tocomplete a message (i.e., expecting more chunks). The mes- sage streamis aborted.

The embodiments of the present invention obtain benefits from breakingthe individual messages further down into chunks. While it may appearmore simple to send one message and then the next, a linear process,such as this, is actually much less efficient. A stream representingsome media presentation may be comprised of several differentsub-streams. One sub-stream may be for the audio, one for the video, andanother for data. The combination of each of these sub-streams makes upthe entire streaming presentation. Therefore, it becomes important tocoordinate smooth delivery of each of the sub-streams to facilitate thesmooth playback of the entire stream. For example, a video key frame mayhave a large amount of data, 20k-30k bytes, in contrast to audiomessages that are generally very small. However, the audio messages aremuch more time sensitive than the video messages due to theidiosyncrasies of human hearing. Therefore, if an audio message is sentfrom the audio sub-stream, followed by a very large video message fromthe video sub-stream, followed by the next audio message, the audiostream may typically stall because the next audio message simply may notarrive in time.

In operation, the larger video messages are typically much lessfrequent, even though they may be much larger. Therefore, interleavingother messages in between the video messages is acceptable as long asthe video message eventually arrives in time for the smooth playback ofthe video sub-stream. Considering this balancing process, breaking themessages into chunks that may be more easily balanced and interleavedincreases the efficiency of the streaming. While no one maximum chunksize is best for all situations, a given maximum may be beneficially setin which the video and data messages arrive without too much effect onthe timely arrival of the audio chunks.

In certain embodiments of the present invention, the maximum chunk sizeis set to correspond to the complete length of a standard audio messagein the audio sub-streams in order to ensure timely delivery of thecomplete audio message. For example, the sampling rate for the audiocompression algorithm, Nellymoser, is typically 8 kHz. The 8 kHzsampling typically generates messages that are 128 bytes long.Therefore, in certain embodiments of the present invention that may usethe Nellymoser compression algorithm, the maximum chunk size maypreferably be set to 128 bytes. Other audio compression formats mayproduce messages that are either greater than or less than 128 bytes. Insuch cases, a low-level protocol message could be used to change themaximum chunk size of the applicable embodiments of the presentinvention to ensure an optimal efficiency algorithm.

Balancing the transmission of the chunks in the different mediasub-streams is preferably controlled by a scheduling algorithm thatdirects which chunk of which stream or sub-stream to send next. Eachtype of media may preferably be assigned a specific priority value. Forexample, because audio messages are the most sensitive, they may beassigned the highest priority value. Video and data messages would eachlikely be assigned lower priority values. FIG. 6 is a flowchartdisclosing representative steps of scheduling algorithm 60 used inrepresentative embodiments of the present invention. In step 600, thepriority of each sub-stream (PrioritySB) is summed into a total streampriority (StreamPriority). In step 601, a desired percentage for sendingeach sub-stream within the stream according to formula (1):Desired %=PrioritySB/StreamPriority  (1)

The total number of chunks sent (count) are counted per sub-stream overthe preceding ‘N’ chunks in step 602. ‘N’ represents a schedulingsampling window that may be preset by a user or developer. In step 603,the actual percentage of chunks sent per sub-stream in the preceding ‘N’chunks is developed according to formula (2):Actual %=count/N  (2)

In step 604, a ratio of desired percentage to actual percentage isdetermined for each sub-stream. A selection criteria, which selects thesub-stream with lowest ratio is executed in step 605. In step 606, aqueue monitor searches or determines whether the selected sub-stream hasa chunk queued. In step 607, if a chunk is queued for the selectedsub-stream, a queue manager selects the sub-stream chunk to send. If nodata is queued for the selected sub-stream. The determination is madewhether there are any other waiting sub-streams left, in step 608. Ifthere are no remaining sub-streams, then no chunks are sent in thecycle, in step 609. However, if there are remaining sub-streams, a newsub-stream is selected, in step 610, that has the next lowest ratio ofdesired to actual percentages. Steps 606-610 are repeated until allchunks in each sub-stream are sent.

Scheduling algorithm 60 preferably results in a favorable balance ofstreams. If all sub-streams always have data to send, not only doesscheduling algorithm 60 provide for those streams to obtain the properpercentage of chunks sent, but also provides a good interleaving mixover the scheduling window. When some sub-streams do not have data or atleast, do not have data queued, scheduling algorithm 60 allows the lowerpriority sub-streams to use a greater bandwidth. Thus, bandwidth use isroutinely optimized. One embodiment of the present invention may achievedesired results with audio having a priority of 100, data priority of10, and video priority of 1. However, other various priority assignmentsmay prove better for different situations and known or expected datatypes or data loads.

FIG. 7A is a conceptual block diagram illustrating real-timecommunication system 70 configured using a representative embodiment ofthe real-time priority-based protocol of the present invention. Streampresentation 700, which may include video sub-stream 704 for providingvisual rendering 701, data sub-stream 705 for providing data visual 702,and audio sub-stream 706 for supporting the audio for visual rendering701, is intended to be streamed from computer 707 to client 712 in apeer-to-peer application. Stream 703 includes each of video, data, andaudio sub-streams 704-706. Computer 707, which includes a processorcapable of running the real-time priority-based protocol describedherein, breaks the video, data, and audio messages into chunks accordingto a maximum chunk size preferably set to accommodate the sampling rateof the audio information. As video, data, and audio sub-streams 704-706are broken into the appropriated chunk size, the necessary headerinformation is placed on each chunk according to the teachings describedherein at state machine 713. The chunks are then queued in chunk queue709 awaiting the determination of which chunks to send. Sub-queues foraudio, video, and data chunks may each be maintained for more efficientdeterminations of the chunks. Scheduler 708 implements a schedulingalgorithm, such as scheduling algorithm 60 (FIG. 6), to schedule andprioritize the transmission of each of the chunks for the differentsub-streams.

It should be noted that if a stream were being received by computer 707instead of transmitting the stream, chunk assembly 718 would be used toassemble the received chunks into the resulting message stream.

The example embodiment shown in FIG. 7A also includes single socketconnection 710 between Internet 711 and both of computer 707 and client712. Scheduler 708 controls which chunks are sent in order toaccommodate delivering stream 703 to client 712, in such a manner that auser at client 712 may view, in real-time, stream presentation 700 witha minimum amount of interference or disturbance in the playback oversingle socket connection 710. As the data chunks are received, chunkassembly 718 on client 712 operates with instructions from state machine713 within client 712 to reassemble the received chunks intopresentation stream 714 with video, audio, and data sub-streams 715-717.It should be noted that instances of state machine 713 are preferablyincluded in each entity compatible with the real-time, priority-basedcommunication system of the present invention. Furthermore, if client712 were transmitting the message stream instead of receiving the chunkstream, queue 709 and scheduler 708 on client 712 would be used totransmit the chunks associated with the message stream.

FIG. 7B is a conceptual block diagram illustrating real-timecommunication system 71 configured using a representative embodiment ofthe real-time priority-based protocol of the present invention in aclient-server application. Similar to the example shown in FIG. 7 A,stream presentation 720, which may include video sub-stream 726 forproviding visual rendering 721, data sub-stream 725 for providing datavisual 722, and audio sub-stream 724 for supporting the audio for visualrendering 721, may originate from computer 727 and viewed on entity 737through server 728 in a client-server topology.

The processing capabilities of computer 727 divides video, audio, anddata sub-streams 724-726 into chunks for streaming over the connectionto server 728. As the chunks are assembled, state machine 713 providethe necessary header information for each chunk as the chunks are queuedin chunk queue 709. Scheduler 708 executes a scheduling algorithm todetermine which chunks should be transmitted in which order. It shouldbe noted that, as above, if a stream were being received by computer 727instead of transmitting the stream chunk assembly 718 would be used toassemble the received chunks into the resulting message stream. Throughconnection 72, which may be the Internet, wide area network (WAN), localarea network (LAN), wireless local area networks (WLAN), or the like,chunks are transmitted to server 728. Using an instance of state machine713 on server 728, the chunks may be received and reassembled, at server728, using chunk assembly 718. Stream 732 at server 728 is comprisedalso of audio, video, and data sub-streams 729-731. Server 728 acts asthe middle-ware providing stream presentation 720 to each entitysubscriber, such as entity 737. As such, when stream 732 is to be sentto client 737, scheduler 708 and queue 709 on server 728 facilitatestreaming the chunks via connection 72.

State machine 713 at server 728 assists the streaming of the chunks ofaudio, video, and data sub-streams 729-731 using connection 72 to entity737. Upon receiving the chunks, state machine 713 at entity 737 helpsdecipher and assemble the header invention with chunk assembly 719.Audio, video, and data sub-streams 733-735 of stream 736 are thenassembled and may be displayed at entity 737. In such an embodiment,while stream presentation 720 may likely be blocked by a firewall or NATfrom reaching entity 737 directly. This complication is preferablyavoided by using the middle-ware functionality of server 728. Server 728preferably has authorized access to entity 737 though any given firewallor NAT. Furthermore, if entity 737 were transmitting the message streaminstead of receiving the chunk stream queue 709 and scheduler 708 onentity 737 would be used to transmit the chunks associated with themessage stream.

When implemented in software, the elements of the present invention areessentially the code segments to perform the necessary tasks. Theprogram or code segments can be stored in a computer readable medium ortransmitted by a computer data signal embodied in a carrier wave, or asignal modulated by a carrier, over a transmission medium. The “computerreadable medium” may include any medium that can store or transferinformation. Examples of the computer readable medium include anelectronic circuit, a semiconductor memory device, a ROM, a flashmemory, an erasable ROM (EROM), a floppy diskette, a compact diskCD-ROM, an optical disk, a hard disk, a fiber optic medium, a radiofrequency (RF) link, etc. The computer data signal may include anysignal that can propagate over a transmission medium such as electronicnetwork channels, optical fibers, air, electromagnetic, RF links, andthe like. The code segments may be downloaded via computer networks suchas the Internet, Intranet, and the like.

FIG. 8 illustrates computer system 800 adapted to use the presentinvention. Central processing unit (CPU) 801 is coupled to system bus802. The CPU 801 may be any general purpose CPU, such as anINTERNATIONAL BUSINESS MACHINE (IBM) POWERPC™, INTEL™ PENTIUM™-typeprocessor, or the like. However, the present invention is not restrictedby the architecture of CPU 801 as long as CPU 801 supports the inventiveoperations as described herein. Bus 802 is coupled to random accessmemory (RAM) 803, which may be SRAM, DRAM, or SDRAM. ROM 804 is alsocoupled to bus 802, which may be PROM, EPROM, EEPROM, Flash ROM, or thelike. RAM 803 and ROM 804 hold user and system data and programs as iswell known in the art.

Bus 802 is also coupled to input/output (1/0) controller card 805,communications adapter card 811, user interface card 808, and displaycard 809. The I/0 adapter card 805 connects to storage devices 806, suchas one or more of a hard drive, a CD drive, a floppy disk drive, a tapedrive, to the computer system. The 1/0 adapter 805 would also allow thesystem to print paper copies of information, such as documents,photographs, articles, etc. Such output may be produced by a printer(e.g. dot matrix, laser, and the like), a fax machine, a copy machine,or the like. Communications card 811 is adapted to couple the computersystem 800 to a network 812, which may be one or more of a telephonenetwork, a local (LAN) and/or a wide-area (WAN) network, an Ethernetnetwork, and/or the Internet network. User interface card 808 couplesuser input devices, such as keyboard 813 and pointing device 807 to thecomputer system 800. The display card 809 is driven by CPU 801 tocontrol the display on display device 810.

FIG. 9 illustrates a sequence of interleaved audio and video chunks inreal-time priority-based communication system 90 configured according tovarious embodiments of the present invention. System 90 shows a one-waycommunication session of streaming media occurring from transmittingentity 91 to receiving player 92 over connection 900. The messages to bestreamed may comprise audio, video, and data messages. FIG. 9 shows amessage that comprises video and audio sub-streams. As the video andaudio sub-streams are broken into video and audio chunks, the statemachine of the various embodiments of the present invention supply thenecessary header information for the chunks. Transmitting entity 91places the chunks in either video queue 93 or audio queue 94 to awaittransmission.

The scheduling algorithm run by transmitting entity 91 controls whenchunks are sent to connection interface 97 from either video or audioqueue 93 and 94. Chunks 901, 903, 905, 907, 909, 911, 913, and 915 areillustrated on connection 900 along with corresponding header parameters902, 904, 906, 908, 910, 912, 914, and 916 configured according to theillustrated example of the present invention. Because audio chunks aremore sensitive than video chunks, the scheduler sends audio chunk 901over connection 900 first. Audio chunk 901 includes header 902. Header902 describes audio chunk as a Type 0 chunk and gives the necessaryinformation that accompanies a Type 0 new message chunk, including thechunk ID, message stream ID, length, message type, and time delta. Thestate machine at receiving player 92 sets the state for the audiosub-stream beginning with audio chunk 901. Audio chunk 903 is the nextchunk sent by the scheduler at transmitting entity 91 and includesheader 904. Header 904 identifies chunk 903 as a Type 3 continuationheader belonging to chunk ID 1, the chunk ID assigned to the first audiosub-stream. Because audio chunk 903 is a continuation chunk, only thechunk type and chunk ID are required in the header information. Thestate machine at receiving entity 92 reads that chunk 903 is a Type 3continuation chunk and keeps all states fixed for the audio sub-streamof chunk ID 1.

The scheduler at transmitting entity 91 sends video chunk 905 next alongwith header 906. Because video chunk 905 is the first of the videosub-stream, it is designated a Type 0 chunk and includes all of thenecessary header information to go along with the first chunk of thevideo sub-stream. Thus, video chunk 905 may include a different chunkID, message stream ID, length, message type, and time delta. The statemachine at the receiving entity sets the necessary states for the videosub-stream of chunk ID 2. The next chunk sent by the scheduler is audiochunk 907 and header 908. Header 908 identifies chunk 907 as a Type 3continuation chunk of chunk ID 1. The state machine refers to the statesset for the audio sub-stream of chunk ID 1 and maintains those settingsfor the first audio sub-stream.

The scheduler then sends audio chunk 909 and header 910 acrossconnection 900. Header 910 identifies audio chunk 909 as a Type 0 newmessage chunk. The state machine at receiving entity 92 resets allinformation in the states of the audio sub-streams and sets the newstates according to header 910. The next chunk in the interleavedsequence is video chunk 911 along with header 912. Header 912 identifiesvideo chunk 911 as a Type 3 continuation chunk of video sub-stream chunkID 2. The state machine at receiving entity 92 accesses the states setfor chunk ID 2 and keeps all of the state information intact for videochunk 911. The next chunk sent on connection 900 is audio chunk 913 andheader 914. Header 914 identifies chunk 913 as a Type 3 continuationchunk of chunk ID 3. The state machine accesses the states saved for theaudio sub-stream of chunk ID 3 and maintains each state intact. Thefinal chunk illustrated in FIG. 9 is audio chunk 915 and header 916.Header 916 identities chunk 915 as a Type 2 chunk representing a newmessage having a new time delta. The state machine reads that chunk 915is a Type 2 chunk and expects the beginning of a new audio sub-streamwith only a change to one of the states for a new chunk ID 4.

As each of chunks 901, 903, 905, 907, 909, 911, 913, and 915 arereceived at receiving entity 92 through connection interface 97, theaudio chunks are re-assembled in audio assembly 96, while the videochunks are re-assembled in video assembly 95 before being played back atreceiving entity 92. The state machine at receiving entity 92 preferablyreads the parameters in each of headers 902, 904, 906, 908, 910, 912,914, and 916 controls the reassembly of the chunks in audio assembly 96and video assembly 95. In additional embodiments of the presentinvention other interleaving scheduling algorithms and/or efficiencyalgorithms may be implemented. The goal is to achieve an efficient mixof audio, video, and data chunks that results in a good playback thatminimizes the audio or video jitter in the playback.

Although the present invention and its advantages have been described indetail. It should be understood that various changes, substitutions andalterations can be made herein without departing from the spirit andscope of the invention as defined by the appended claims. Moreover, thescope of the present application is not intended to be limited to theparticular embodiments of the process, machine, manufacture, compositionof matter, means, methods and steps described in the specification. Asone of ordinary skill in the art will readily appreciate from thedisclosure of the present invention, processes, machines, manufacture,compositions of matter, means, methods, or steps, presently existing orlater to be developed that perform substantially the same function orachieve substantially the same result as the corresponding embodimentsdescribed herein may be utilized according to the present invention.Accordingly, the appended claims are intended to include within theirscope such processes, machines, manufacture, compositions of matter,means, methods, or steps.

Having thus described the invention, what is claimed is:
 1. Acomputer-implemented method, the method comprising: obtaining aplurality of sub-streams associated with a media stream, each of thesub-streams divided into a plurality of chunks; providing header datafor each chunk of the plurality of chunks, wherein the amount of headerdata provided for each chunk is based on a chunk type identifierassociated with the chunk that indicates an extent of header datacontained in a prior chunk that corresponds with the chunk such thatchunks have variable header lengths; and determining a schedule fortransmitting the plurality of chunks, wherein the schedule balancestransmission of the chunks from the plurality of sub-streams inaccordance with a real-time priority value assigned to each of thesub-streams of the media stream based on a lowest ratio of a desiredtransmission rate over an actual transmission rate calculation.
 2. Themethod of claim 1, wherein the plurality of sub-streams comprise anaudio sub-stream and a video sub-stream.
 3. The method of claim 1,wherein the plurality of sub-streams comprise a video sub-stream and adata sub-stream.
 4. The method of claim 1, wherein the plurality ofsub-streams comprise an audio sub-stream and a data sub-stream.
 5. Themethod of claim 1, wherein the chunk type identifier indicates a newstream chunk, a new media chunk, a new time chunk, or a continuationchunk.
 6. The method of claim 1 further comprising queuing the chunks ina chunk queue for transmission to a remote computer.
 7. The method ofclaim 1, wherein determining the schedule comprises: assigning apriority to each of the one or more sub-streams; totaling a priority forthe media stream; calculating the desired transmission rate for each ofthe one or more sub-streams using the assigned priority and the totaledpriority; counting a sub-total of the plurality of associated chunkstransmitted per the one or more sub-streams during a preset period;counting a total number of the plurality of associated chunkstransmitted during the preset period; calculating the actualtransmission rate for each sub-stream using the sub-total and the total;and calculating the ratio between the desired transmission rate and theactual transmission rate.
 8. A computer-implemented method comprising:obtaining chunks associated with a first media sub-stream of a firstmedia type and chunks associated with a second media sub-stream of asecond media type, the first media sub-stream and the second mediasub-stream corresponding with a media stream; determining a schedule fortransmitting the plurality of chunks, wherein the schedule balancestransmission of the chunks from the first media sub-stream of the firstmedia type and chunks from the second media sub-stream of the secondmedia type in accordance with a first real-time priority value assignedto the first media sub-stream and a second real-time priority valueassigned to the second media sub-stream based on a lowest ratio of adesired transmission rate over an actual transmission rate calculation;and transmitting the chunks associated with the first media sub-streamof the first media type and the chunks associated with the second mediasub-stream of the second media type in accordance with the determinedschedule.
 9. The method of claim 8, wherein the first real-time priorityvalue and the second real-time priority value are assigned based on thecorresponding media type.
 10. The method of claim 8, wherein theschedule indicates whether to transmit a chunk from the first mediasub-stream or the second media sub-stream.
 11. The method of claim 8,wherein the first media type comprises an audio media having a higherpriority value, and the second media type comprises a video media havinga lower priority value.
 12. A system comprising: a storage device; andone or more computer systems communicatively coupled with the storagedevice and operable to interact with a user device, the one or morecomputer systems being configured to perform operations including:obtaining chunks associated with a first media sub-stream of a firstmedia type and chunks associated with a second media sub-stream of asecond media type, the first media sub-stream and the second mediasub-stream corresponding with a media stream; determining a schedule fortransmitting the plurality of chunks, wherein the schedule balancestransmission of the chunks from the first media sub-stream of the firstmedia type and chunks from the second media sub-stream of the secondmedia type in accordance with a first real-time priority value assignedto the first media sub-stream and a second real-time priority valueassigned to the second media sub-stream based on a lowest ratio of adesired transmission rate over an actual transmission rate calculation;and transmitting the chunks associated with the first media sub-streamof the first media type and the chunks associated with the second mediasub-stream of the second media type in accordance with the determinedschedule.
 13. The system of claim 12, wherein the first real-timepriority value and the second real-time priority value are assignedbased on the corresponding media type.
 14. The system of claim 12,wherein the schedule indicates whether to transmit a chunk from thefirst media sub-stream or the second media sub-stream.
 15. The system ofclaim 12, wherein the first media type comprises an audio media having ahigher priority value, and the second media type comprises a video mediahaving a lower priority value.